Skip to content

ZeRO-3: take the AutoTP shard from universal checkpoints converted without TP - #9069

Merged
pengdurice merged 2 commits into
deepspeedai:masterfrom
Achyuthan-S:zero3-uc-plain-tensor-autotp
Nov 17, 2026
Merged

pengdurice merged 2 commits into
deepspeedai:masterfrom
Achyuthan-S:zero3-uc-plain-tensor-autotp

Conversation

@Achyuthan-S

@Achyuthan-S Achyuthan-S commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

A ZeRO-3 checkpoint saved without TP can't be resumed in an AutoTP + ZeRO-3 run through the universal checkpoint.

Why

ds_to_universal writes each state of a ZeRO-3 checkpoint saved without TP as a bare tensor (merge_zero3_slices); every other path writes a dict with the tensor under param. DeepSpeedZeroOptimizer_Stage3.load_hp_checkpoint_state takes the AutoTP shard only when the file holds a dict. With a bare tensor it partitions the whole weight as if it were this rank's TP shard, and the copy into the flat buffer fails:

RuntimeError: start (272) + length (256) exceeds dimension size (280).

_resolve_autotp_partition cuts the shard using the target parameter's AutoTP metadata and the loaded tensor, and never reads the dict, so the check isn't needed.

What

The target parameter alone decides whether to slice (if param is not None:). Checkpoints converted before this keep loading, since the loader accepts both formats as before.

Whether every conversion path should write the same format is left for later, as discussed with @delock: the loader would still have to accept bare tensors from checkpoints that were already converted.

No open PR touches this check (searched for load_hp_checkpoint_state and ZeRO-3 universal AutoTP; #8848 changes the converter, not stage3.py).

Test

TestZero3UniversalCheckpointWithoutTp in test_autotp_uc_checkpoint.py trains 4 steps under ZeRO-3 without TP, converts, and loads into ZeRO-3 at TP1 and at AutoTP TP2. It compares the fp32 weights, both Adam moments and the step counts with the saved state bit for bit, since conversion and restore only copy. On master the TP2 case fails with the error above; the TP1 case passes before and after. _affine_resume_engine gains a zero_stage argument (default 1) so the test reuses the existing resume model and helpers.

Validation

  • GPU, 4× A100-SXM4-40GB, driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7, Python 3.12.11, pytest --forked tests/unit/checkpoint/test_autotp_uc_checkpoint.py tests/unit/checkpoint/test_universal_checkpoint.py tests/unit/checkpoint/test_autoep_affine_universal.py: 217 passed with the fix. With only the stage3.py change reverted, the new TP2 case fails with the error above and the TP1 case passes.
  • CPU, macOS arm64, Python 3.9.6, torch 2.8.0, gloo (DS_ACCELERATOR=cpu LOCAL_SIZE=4): the same result for the new test, before and after. The FusedAdam tests in these files can't build the op on macOS, so they ran on GPU only.
  • Qwen3-0.6B, bf16, the same 4× A100 node type and software, DP4 → TP2×DP2 → DP4 under ZeRO-3, 20 steps per layout. Each change saves, runs ds_to_universal and loads in the new layout, then compares every fp32 weight and both Adam moments with the saved state bit for bit. On master the TP2×DP2 load fails with start (187907840) + length (1048576) exceeds dimension size (187924480); with the fix both changes restore all 930 tensors bit for bit.
  • pre-commit passes on the changed files.

Checked and not changed

  • ZeRO-1/2 (universal_checkpoint.load_hp_checkpoint_state) reads ckpt_dict[PARAM] unconditionally, but stage 1/2 conversion always writes the dict, and a ZeRO-3 universal checkpoint doesn't reach that loader today: a ZeRO-1 engine looks for mp_rank_00_model_states.pt, while the converted ZeRO-3 folder has zero_pp_rank_0_mp_rank_00_model_states.pt, so the load stops in state_dict_factory.check_ckpt_list.
  • The AutoEP branch above this check is unchanged.

Validated: ZeRO-3 without offload; fp32 (the unit test) and bf16 (Qwen3-0.6B); TP1 → TP1, TP1 → TP2, and DP4 → TP2×DP2 → DP4. Not validated: CPU or NVMe offload, fp16, pipeline parallelism.

…thout TP

ds_to_universal writes each state of a ZeRO-3 checkpoint saved without TP as a bare tensor, while every other path writes a dict. The ZeRO-3 loader resolved the AutoTP shard only for a dict, so loading such a checkpoint into an AutoTP run partitioned the whole weight and failed. _resolve_autotp_partition reads only the target parameter's metadata and the loaded tensor, so let the target decide.

Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 20:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T20:23:36.161084Z f4e3b11 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Achyuthan-S

Copy link
Copy Markdown
Contributor Author

A real-model check of the same change: Qwen3-0.6B, bf16, 4× A100 (driver 595.71.05, torch 2.12.1+cu130, NCCL 2.29.7), DP4 → TP2×DP2 → DP4 under ZeRO-3, 20 steps per layout. Each change saves, runs ds_to_universal and loads in the new layout, then compares every fp32 weight and both Adam moments with the saved state bit for bit.

  • master (988b2e3): the TP2×DP2 load fails with RuntimeError: start (187907840) + length (1048576) exceeds dimension size (187924480).
  • this PR: both changes restore all 930 tensors bit for bit.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 17, 2026
…time (deepspeedai#8844)

# Universal Checkpointing: read AutoEP ZeRO-3 optimizer files one at a
time

## Problem

`ds_to_universal` converts a ZeRO-3 AutoEP checkpoint in two parts. The
dense parameters go through the usual
extract-and-merge path, which reads one optimizer file per worker and
writes fragments to `tmp/`. The routed experts
go through `_consolidate_zero3_autoep_expert_states`, which first loads
**every** rank's optimizer file into one dict
(`optim_states_by_rank`) and then keeps a copy of every expert fragment
in memory until all of them are merged.

Together those files hold every optimizer state of the model: fp32 (or
bf16) master weights plus both Adam moments.
For a large MoE model that is more than one host has:

| Model | Parameters | Optimizer state, fp32 master + fp32 Adam (12
B/param) | bf16 master + bf16 Adam (6 B/param) |
|---|---|---|---|
| MiniMax-M3 | 426.2B | 4.65 TiB | 2.33 TiB |
| GLM-5.2 | 743.4B | 8.11 TiB | 4.06 TiB |

The H200 nodes this was found on have 2 TiB of host memory, so the
conversion cannot finish for either model.

## Change

`_consolidate_zero3_autoep_expert_states` reads one rank's optimizer
file at a time:

- Each expert fragment it finds is written to
`<output>/tmp/autoep_expert_fragments/` (one file per parameter,
  optimizer state, EP rank and partition rank).
- Every reference into the rank's optimizer state, including the loop
variables, is dropped before the next file is
  read.
- The merge then loads, for one parameter and one state at a time, only
that parameter's fragments, and deletes each
fragment file once it is read, so the disk does not hold every fragment
and every universal file at the same time.

Peak host memory becomes one optimizer file plus one full expert tensor,
instead of all optimizer files.

The checks are unchanged: missing ranks, duplicate or missing
expert-data-parallel partitions, shape mismatches and
placement disagreements raise the same errors. The temporary fragments
live under `<output>/tmp`, which the converter
already uses for the dense fragments and removes at the end.

## Peak host memory

Measured with a probe on one checkpoint: a 4-layer mock MoE model (64
experts, hidden 1024, expert intermediate
4096: 3.22B expert parameters) trained one step and saved with ZeRO-3
and AutoEP EP 4 on 8 GPUs (bf16, Adam), giving
8 optimizer files of 36.19 GiB in total. Each version ran only
`_consolidate_zero3_autoep_expert_states`, in its own
process; peak resident memory is `ru_maxrss` (0.4 GiB of it is the
imports).

| | Peak host memory | Time | Output |
|---|---|---|---|
| master 3fe2d95 | 109.20 GiB | 93.6 s | |
| this change | 13.13 GiB | 203.6 s | all 36 universal files (fp32,
exp_avg, exp_avg_sq of each expert weight) bitwise equal to master's |

Master holds about 3 times the optimizer files at once; with this change
the peak is about one optimizer file plus one
parameter's tensors, and it does not grow with the number of ranks. The
cost is time: the fragments are written to
and read back from `<output>/tmp`, 2.2x slower here on a shared Lustre
file system. For MiniMax-M3 and GLM-5.2 the
old path cannot finish at all on a 2 TiB host.

Hardware: 1 node of 8 NVIDIA H100 80GB (driver 595.112.07), torch
2.8.0+cu126. Probe script:
`ucp_expert_memory_probe.py` (modes `save`, `convert`, `compare`).

## Tests

On master 3fe2d95 with this change, 1 node of 8 NVIDIA H100 80GB
(driver 595.112.07), torch 2.8.0+cu126, pytest 8.3.5:

| Test | Result |
|---|---|
| `tests/unit/checkpoint/test_autoep_affine_universal.py` (calls the
function directly) and `tests/unit/v1/moe/test_autoep_checkpoint.py`
(AutoEP ZeRO-3 save, convert, load at other layouts) | 69 cases: 67
passed, 2 skipped (ZeRO-1/2 universal-checkpoint cases that run only
with `DEEPSPEED_RUN_H100_TESTS=1`; this change is on the ZeRO-3 path) |

`pre-commit run --files` passes on the changed file.

## Duplicate check

Searched open pull requests on 2026-11-13 ("ds_to_universal", "universal
checkpoint AutoEP"). deepspeedai#9069 and deepspeedai#8973 change
how ZeRO-3 loads universal checkpoints and partial-offload checkpoints,
not `ds_to_universal.py`. deepspeedai#8848 (mine) also
edits `_consolidate_zero3_autoep_expert_states` (keys by tensor-parallel
and data-parallel rank); whichever of the two
merges second needs a small rebase.

## Compatibility

| Cell | Validated |
|---|---|
| ZeRO-3 + AutoEP checkpoint conversion, save / convert / load at other
layouts | yes (tests) |
| Dense (non-AutoEP) ZeRO-3 conversion | path unchanged |
| ZeRO-1 / ZeRO-2 conversion | path unchanged |

## AI assistance

This change, its tests and this description were prepared with AI
assistance (Claude).

---------

Signed-off-by: pengdurice <pengduhit@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@gpu-test-labeler gpu-test-labeler Bot removed the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 17, 2026
@pengdurice
pengdurice enabled auto-merge November 17, 2026 19:23
@pengdurice
pengdurice added this pull request to the merge queue Nov 17, 2026
Merged via the queue into deepspeedai:master with commit 5420b75 Nov 17, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants